Skip to content

server : add [DONE] event to /chat/completions stream response #9459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 14, 2024
Merged

server : add [DONE] event to /chat/completions stream response #9459

merged 1 commit into from
Sep 14, 2024

Conversation

VoidIsVoid
Copy link
Contributor

@ggerganov
Copy link
Member

What is the purpose of this?

@VoidIsVoid
Copy link
Contributor Author

VoidIsVoid commented Sep 13, 2024 •
edited
Loading

@ggerganov
OpenAI's /chat/completions has a "data: [DONE]\n\n" at the end of stream.
Some openai api compatible client just treat it as the end of stream request.
I think it's better to follow openai's behavior.

ngxson
ngxson requested changes Sep 13, 2024
View reviewed changes
Copy link
Collaborator

@ngxson ngxson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could confirm that this is missing from our implementation. It's actually documented and here is an example of request having this [DONE] message:

image

I tried adding this in the past but was having some issues with the test.

Currently, this test case will fail:

Failing scenarios:
  features/parallel.feature:56  Multi users OAI completions compatibility -- @1.2

So you need to update the test script to be able to handle this.

@github-actions github-actions bot added the python python script changes label Sep 13, 2024
@VoidIsVoid VoidIsVoid requested a review from ngxson September 13, 2024 16:00
@ngxson ngxson changed the title server: add data: [DONE] to /chat/completions stream response server : add [DONE] event to /chat/completions stream response Sep 14, 2024
@ngxson ngxson merged commit dcdcee3 into ggml-org:master Sep 14, 2024
53 checks passed
@isaac-mcfadyen
Copy link
Contributor

Wonder if we should add this to the README.md in some place?

This is breaking for many apps that use the stream raw without an OpenAI compatible client (i.e. via ReadableStream).

@ngxson
Copy link
Collaborator

ngxson commented Sep 14, 2024

IMO this is not a breaking change, because it was break before and now we just fix it. Also, /chat/completion is declared as OAI-compatible, so usage without OAI-compatible clients is not expected

This was referenced Sep 15, 2024
Open
Closed
dsx1986 pushed a commit to dsx1986/llama.cpp that referenced this pull request Oct 29, 2024
@VoidIsVoid @dsx1986
server: add data: [DONE] to /chat/completions stream response (ggml-o…
7cd1f07 
…rg#9459)
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 15, 2024
@VoidIsVoid @arthw
server: add data: [DONE] to /chat/completions stream response (ggml-o…
b8f83e3 
…rg#9459)
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 18, 2024
@VoidIsVoid @arthw
server: add data: [DONE] to /chat/completions stream response (ggml-o…
7d4e141 
…rg#9459)
@CentricStorm CentricStorm mentioned this pull request Nov 29, 2024
Merged
4 tasks
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Feb 25, 2025
@VoidIsVoid @Nexesenex
server: add data: [DONE] to /chat/completions stream response (ggml-o…
e0e5eae 
…rg#9459)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngxson ngxson ngxson approved these changes

Assignees
No one assigned
Labels
examples python python script changes server
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@VoidIsVoid @ggerganov @isaac-mcfadyen @ngxson